When something is out of the way, it's not easy to get to. You could describe your house on an island as out of the way.
Your brother might refuse to stop at the library and pick up a book for you if it's out of the way, while he might love spending weekends at your family's out of the way cabin in the woods. When an errand or visit takes you far off your regular route, your trip is out of the way. Another way to say out of the way is distant or remote.